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Context 


PR3-4 


Background 
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Objectives  of  Assessment 


•  Comparison  of  alternate  systems  or  solutions 

-  replacement  systems  or  components 

-  determination  of  most  cost-effective  approaches 

-  assessment  in  new  or  unexpected  applications 

•  Establishment  of  standards,  bounds  of  performance 

•  Identification  of  potential  weaknesses 

•  Analysis  of  effectiveness  of  training 

•  Evaluation  of  effectiveness  of  human  decision  making 

•  Assistance  in  requirements  generation  and  validation 


PR3-6 


MoM  Definitions 


•  DP  -  Dimensional  Parameters 

-  Properties  or  characteristics  in  physical  entities 

•  MoP  -  Measures  of  Performance 

-  Measures  of  attributes  of  internal  system  behaviour 

•  MoCE  -  Measures  of  C2  Effectiveness 

-  Measures  impact  of  C2  systems 

•  MoFE  -  Measures  of  Force  Effectiveness 

-  Measures  of  how  a  force  meets  mission  objectives 

•  MoPE  -  Measures  of  Policy  Effectiveness 
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MoM  Hierarchy 

Environment 
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Linked  MoMs 
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MoM  Tendencies 
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Characteristics  of  Measures  (1 ) 


•  Reliability 

-  accuracy  of  a  measurement:  variance  of 
repeated  measurements  of  the  same 
phenomenon 

-  must  be  known  or  estimated  to  discriminate 
between  real  effects  and  measurement  effects 
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Characteristics  of  Measures  (2) 


•  Validity 

-  internal:  causal  relationship  between  variables 

-  construct:  measure  objective,  and  only  objective 

-  statistical  conclusion:  results  are  robust  with 
sufficient  sensitivity 

-  external:  extent  to  which  results  could  be 
generalized 

-  expert:  degree  accepted  by  experts  in  the  field 
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Levels  of  Evaluation 


•  Goals  (mission  objectives)  -  Environment 

•  Functions  and  sub  functions 

•  Tasks 

•  Structure  /  Interfaces 

•  Physical  Entities 
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Example  HQ  MoMs  -  Levels 


•  Network  of  headquarters 

•  Single  headquarters 

•  Cells  within  the  HQ 

•  Specific  tasks  within  cells 
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Example  HQ  MoMs 


•  Monitoring  and  understanding 

-  Information  transmission,  values,  times, 
effect,  comprehension 

•  Planning 

-  Information  exchange,  co-ordination, 
impact,  flexibility,  process  quality 

•  Directing  and  disseminating 
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Categories  of  Performance  Measures 


ORGANIZATION 


•  Time  based 

-  time  to  perform  a  task 

-  rate  of  performing  tasks 

-  time  to  react  to  events 

•  Accuracy  based 

-  precision  of  performance 

-  reliability  of  performance 

-  completeness 

-  error  rates 

-  quality  of  decisions 
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CD 


Collaboration  Metrics 


ORGANIZATION 


•  Averages  of  understanding  among  team 
members 

•  Extent  of  alignment  of  these  understandings 

•  Maximum  level  of  understanding  within  team 

•  Gaps  in  understanding  throughout  team 
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Normality  Indicators 


•  Relative  measures 

•  State  of  normalcy 

•  Characterize  an  element  of  the  civil 
environment 

•  Data  collected  on  a  regular  basis 

•  Assessment  of  the  changes 
occurring  in  the  civilian  populace 
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Normality  Indicators 


Criterion 

Examples 

Political 

Elections,  political  participation 

Economic 

Unemployment,  interest  rates,  and  market  baskets 

Social 

Number  of  students  in  schools,  number  of  refugees 

Technological 

Telephone  system  availability 

Legal 

Judicial  system  functioning 

Environmental 

Roads,  water  supply,  power  supply 

Cultural 

Sports  events,  concerts 
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Limitations  of  Normality  Indicators 

ORGANIZATION 

•  Inexperienced  personnel 

•  Limited  resources,  constraints 

•  Effect  of  military  presence 

•  Require  data  to  be  calibrated  against  baselines 

•  Extrapolation  across  space  and  time 

•  Shifting  emphasis,  thresholds 


PR3-21 


Effects  of  Uncertainty 


•  Study  assumptions 

-  uncertainties  in  scenario,  model  input 

•  Modelling  assumptions 

-  Uncertainties  in  the  model,  structural  uncertainty 

•  Model  sensitivity 

-  Uncertainties  in  the  outcome 
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Summary :  Framework 

ORGANIZATION 

•  Establish  evaluation  environment 

•  Define  evaluation  goals 

•  State  context,  assumptions,  constraints 

•  Define  domain  -  MoPE,  MoFE,  MoCE,  MoP,  DP 

•  Identify  specific  measures 

•  Establish  scenario  or  stimulus 

•  Establish  data  collection  means 

•  Pilot  test,  revise  measures  and  procedures 

•  Conduct  the  tests,  debrief  and  analyze 
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Challenges  /  Issues 


•  Linkage  of  DP-M0P-M0CE-M0FE 

•  Interpretation  of  measures 

•  Environmental  components 

•  Reliability  and  validity 

•  Uncertainties  -  scenario,  model,  outcomes 

•  Human-in-the-loop 

•  Cost  and  convenience 

•  Modelling 
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CD 


Recommendations 


ORGANIZATION 


•  Plan  with  clear  objectives 

•  State  assumptions,  constraints 

•  Formally  assess  reliability  and  validity 

•  Concentrate  on  MoCE  and  MoP 

•  Incorporate  MoM  data  gathering  into  system  design 

•  Include  Subject  Matter  Experts  in  assessments 

•  Retain  data  as  benchmarks  for  future  comparison 
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Concluding  Remarks 


•  No  single  measure  or 
methodology  exists  for  assessing 
overall  effectiveness  of  C&C 

•  A  multi-method,  multi-phase 
approach  is  necessary 
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